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Abstract 


Computational thinking is recognized as a vital skill related to problem-solving in technological and 
non-technological fields. The existence of different sub-domains related to this skill has been pointed out. 
Therefore, there is a need for tools that measure these different sub-domains. Because of its structure that 
includes different skills, computational thinking has a structure different from that of the tools used to 
measure academic skills. Moreover, no special programming knowledge is required for tools that measure 
this ability. In order to measure this skill in younger age groups, it is possible to apply the measurement 
tool without adult support. At this point, it is aimed to reveal the computational thinking skills of Turkish 
children by adapting a test developed for the 7-9 age groups into Turkish. For this purpose, an adaptation 
research study was performed for TechCheck-2 developed by Relkin et al. (2020). In the study, a total of 
372 primary school students studying in Ankara were contacted. Item and test analyses were performed 
on the data obtained as a result of the application of the test. The distinctiveness and difficulty values of 
the items making up the test and Kuder Richardson-20 scores were calculated. At the end of the analyses, 
it was seen that the test could be used as a valid and reliable measurement tool for Turkish children. 
Keywords: computational thinking, item analysis, test adaptation, primary school student 


Introduction 


Humankind is in a massive technological transformation. During this transformation, 
people who are integrated with the industry 4.0 revolution will have to produce solutions to new 
events and problems that they have never encountered before, and to continue their existence 
by thinking in new contexts and systems. While some jobs will be disappeared from the face 
of the earth, futurists emphasize the importance of individuals making decisions that determine 
their career goals and future decisions and developing themselves in line with these decisions. 
In this respect, today's educational institutions should adopt a proactive approach, educate 
individuals in accordance with their future vision, and equip them with the necessary skills. 
Computational thinking (CT) can be considered one of these skills (Denning, 2009) because 
as the importance of problem-solving skills such as abstraction, decomposition, algorithmic 
design, generalization, and evaluation increases, the interest in developing CT thinking in 
schools has started to increase (Voogt et al., 2015). This interest stems from the need to learn 
digital languages, which are necessary to be successful in the digital world, to learn coding as 
a way of solving problems and CT as a working paradigm (Garcia-Pefalvo & Mendes, 2018). 
This emerging need indicated that CT thinking skills should be gained from an early age. 

According to Wing (2008), CT skill is an approach that incorporates computational 
fundamental concepts for solving problems, designing systems, and understanding human 
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behaviour. This approach is a set of ideas that allow individuals to understand the world around 
them (Brennan & Resnick, 2012; Chalmers, 2018). Denning and Tedre (2019) considered some 
mental practices necessary in order to put these ideas into practice. The International Association 
for Technology in Education (ISTE, 2011) states that CT is a problem-solving process in which 
formulations are made using computers and other tools, emphasizing that it is a combination of 
organizing and analysing data that puts data in a logical order and automating solutions through 
algorithmic thinking. D’Alba and Huett (2017) have stated that in order to understand CT, it is 
necessary to distinguish between what computers can do better than humans and what humans 
can do better than computers, and that problems can be solved only by designing efficient 
systems in this way. Shute et al. (2017) examined the models in the CT-related literature and 
defined CT as the conceptual basis needed to solve problems effectively and efficiently (e.g., 
with or without computer aid with algorithms) including solutions that can be used in different 
contexts. Berland and Lee (2011) have considered CT within five categories (conditional logic, 
algorithm building, debugging, simulation, and distributed computing) and two stages (local 
logic and global logic). In the literature, it is seen that algorithmic thinking, debugging, and 
parsing dimensions are frequently mentioned (Tosik Giin & Giiyer, 2019; Uziimcti & Bay, 
2018). Grover (2021) has stated that CT is an important tool for participation in the 21st 
century career opportunities and cultural context in both academic and non-academic settings. 
Therefore, maximizing students' CT skills is important for their future. That is why qualified 
measurement tools are needed to determine how successful children are in CT activities. Many 
researchers have stated in their studies that there are deficiencies in measuring CT skills with 
assessment-evaluation tools suitable for age group and education level, and that the number 
of valid and reliable measurement tools for CT is low (Tosik Giin & Giiyer, 2019). Top and 
Arabacioglu (2021), in their research in which they analysed studies on CT, determined that 
Likert-type scales were mostly used to measure this skill in research. Assessments based on the 
person’s own evaluations may be insufficient to measure CT skill. In addition, tools measuring 
skills in different categories related to CT have not yet become widespread enough (Eguchi & 
Urive, 2009). Research on CT in Turkey have focused mostly on secondary school (Cetin et al., 
2020; Giiler & Dinci, 2019; Ibili & Giinbatar, 2020; Paf, 2019), high school (Bulut & Yilmaz, 
2021; Ibili et al., 2020), and university students (Akgiin, 2020; Yel, 2021). Studies focusing on 
CT skills in younger age groups are relatively rare. This situation can be explained by the lack 
of valid and reliable measurement tools developed on this subject. Therefore, the development 
of tools that measure CT skills or the adaptation of tools with international applications to 
Turkish will support the research in this field. For this reason, in the present research, it is 
aimed to reveal the CT skills of Turkish children by adapting the CT test (Relkin et al., 2020) 
developed for the 7-9 age groups into Turkish. 


Computational Thinking Test for Primary School Students 


Relkin et al. (2020) have developed tests for different age groups entitled "TechCheck". 
According to this, 

e TechCheck-K is best for children 5-6 years old (kindergarten) 

e TechCheck-1 is best for children 6-7 years old (first grade) 

e TechCheck-2 is best for children 7-9 years old (second grade) 

In the present research, an adaptation study on the TechCheck-2 version of the TechCheck 
test was carried out. For TechCheck-2, students can read and answer questions on their own 
without the help of an adult (teacher or parent). The test includes questions in four areas covering 
CT skills. This includes algorithmic thinking (missing symbol series, shortest path puzzles, 
sequencing challenge), modular structure (object decomposition), control structures (obstacle 
mazes), representation (symbol shape puzzles), software/hardware (identifying technological 
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concepts), and debugging. There are different types of questions in the fields of symmetry | 767 
problem-solving. The questions in the test are based on important ideas for CT expressed by 

Bers (2017). The connection between the question contents and the important ideas is presented 

in Table 1. 


Table 1 
CT Test Domains 
Domain Task Type Sample 
Bu tavsan bir seferde yalmizea bir beyaz kare ziplayabilir. 
Tavsanin iki havug almasinin en hizh yolu hangisidir? 
A. B. 
missing symbol 
series, shortest 
i ba th puzzl 
Algorithmic nna 
Thinking ene C D. 
challenge 
Bu kardan adami yapmak icin hangi sekillere ihtiyacin var? 
Modular object A B 
Structure decomposition x ex JOC Te ee 


° O0oO7NO ver 


A. B. =Saee 
Gant obstacle mazes Db eo 


Structures 


The test can be applied face-to-face or with the help of online platforms (e.g., Qualtrics, 
Ispring, pear deck). In the face-to-face application, an average of 12 minutes is sufficient. The 
questions in TechCheck-2 are similar to those in TechCheck-1, but the level of difficulty varies. 
There are 17 questions in total in the test, but the first two questions are not included in the 
scoring (Figure 1). The first two questions at the beginning of the test are included as practice 
round questions. The questions in the test are scored as 1-0 and the total score is calculated. 
There are four options in the multiple-choice test. The items created for the test were analysed 
and evaluated separately, depending on both the item response theory and the classical item 
theory. The Cronbach's alpha value calculated for the original test is .68 (Relkin et al., 2020). 
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Practice Round Questions 


~Aligtirma Turu 1- -Alistirma Turu 2- 


Hangisi bir hayvandir? Hangisi bir hayvandir? 


Research Purpose 


The aim of the present research was to adapt the TechCheck-2 version of the “TechCheck” 
test developed by Relkin et al. (2020) into Turkish. Within the scope of CT skills, a test 
adaptation was carried out at the preschool level, which was previously under the umbrella 
of basic education (Ulker Hancer et al., 2021). However, no measurement tool developed in 
Turkish at the primary school level was encountered. In today's computer-based society, it is 
necessary for children to be computer literate as users or creators of digital technology (Bers & 
Sullivan, 2019). Therefore, the use of a tool with validity and reliability in revealing these skill 
areas of Turkish children will be beneficial for researchers, teachers, and program development 
research. 


Research Methodology 


The TechCheck-2 test developed by Relkin et al. (2020) was adapted into Turkish. In 
this context, the survey model was used. In the survey model, attempts are made to describe an 
existing situation (Cepni, 2010), but the causality is not explained (Biiytik6ztiirk et al., 2012). In 
the present study, researchers tried to reveal an existing situation through a sample taken from 
the population. 


Group 


The participants in the study were 2nd- and 3rd-year students studying in Ankara in 
the 2021-2022 academic year. Relkin et al. (2020) mentioned that the age range in which 
the TechCheck-2 test can be applied is 7-9. Therefore, for the Turkish adaptation, the study 
was carried out with primary school students in the same age group. A convenience sampling 
technique was used to collect the data. The convenience sampling technique is a method that 
accelerates the research because it allows the researcher to act according to the situation that 
is close and easy to access (Etikan et al., 2016). In this context, a total of 372 students (,._...= 
193, n.,,,.— 179) were contacted. While 157 of the students were studying in the 2nd grade, 215 
of them were in the 3rd grade i 7.9, SD = 0.718). 
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Instruments and Procedures 


In this adaptation research, the data were collected with the help of a questionnaire 
developed by the researchers. In the first part of the relevant questionnaire, the gender, age, and 
class information of the students were included. The second part of the questionnaire contained 
the articles of TechCheck-2 translated into Turkish. The final configuration of the questionnaire 
was delivered to primary school 2nd- and 3rd-grade students through classroom teachers after 
necessary permissions were obtained. The questionnaire was sent to the students online and the 
facilities of the schools they studied at were used to fill out the questionnaire. 


Ethical Permissions 


Within the scope of the research, permission to use the TechCheck-2 test was first 
requested from the responsible author. The corresponding author asked the purpose of the test, 
the age group to which it would be applied, and the method to be followed in the analysis 
of the data and gave permission to use TechCheck-2 for the adaptation research in line with 
the purpose of this research. In this context, the scoring method of the test, the points to be 
considered in the application in face-to-face and online environments, the items that make 
up the test, and the answer key were sent to the researchers via e-mail. After obtaining the 
adaptation permissions, the form regarding the ethics committee permission, dated 2021 and 
decision number 573, obtained from the Social and Human Sciences Scientific Research Ethics 
Committee of Necmettin Erbakan University, was applied to the students who voluntarily 
agreed to participate in the research. 


Test Adaptation Process 


After obtaining permission to use the relevant test, the test was first translated into 
Turkish by the researchers. The translation was sent to three experts along with the original. 
Two of the experts have very good English language skills and they are faculty members with 
doctorate degrees. One of the experts is an associate professor who has academic publications 
on CT skills. The experts compared the translated test with the original version and gave their 
opinions. For this, they contributed to the adaptation process by filling in the "appropriate, not 
appropriate, and justification" sections on the form sent to them. The test items translated into 
Turkish were not retranslated. Visuals are more important than expressions in test items, and 
expressions for primary school children should be quite clear and unambiguous. Therefore, it 
was sent to the Turkish teacher, who is doing a doctorate, to check the expressions translated 
into Turkish in terms of grammar and meaning. After obtaining confirmation about the 
appropriateness of the expressions in terms of Turkish, the test was administered to two primary 
school students who were not included in the sample. The aim was to check whether the test 
items were understood. With the feedback from the students, the test was finalized, and the 
application form was prepared. 


Data Analysis 


The CT test is coded as 1-0. Therefore, the item difficulty values (pjx) and item 
discrimination values (rjx) were calculated for this test. Pjx is a measure of the extent to which 
the item is answered correctly, and as the obtained value approaches, the item becomes more 
difficult, and as it approaches 1, the item becomes easier (Tekin, 1991). Rjx, on the other hand, 
is a measure of how well the items distinguish between those who know and those who do not 
know about the measured feature. Accordingly, rjx takes a value between -1 and +1 (Turgut, 


ISSN 1822-7864 (Print) ISSN 2538-7111 (Online) https://doi.org/10.33225/pec/22.80.765 


PROBLEMS 
OF EDUCATION 


IN THE 21*CENTURY 
Vol. 80, No. 6, 2022 


769 


PROBLEMS 
OF EDUCATION 


IN THE 21* CENTURY 
Vol. 80, No. 6, 2022 


770 


Mensure ALKIS KUGUKAYDIN, Gigdem AKKANAT. Adaptation into Turkish of the computational thinking test for primary school 
students 


1992). In the literature (Tekin, 1991; Turgut, 1992), some cut-off points have been reported for 
pjx and rjx (Table 2). Accordingly, items with a value of .40 and above in terms of discrimination 
are considered very good, while items with a value of .19 and below are considered very weak. 
In terms of difficulty, items with a value of .61 and above were reported as easy, and items with 
a value of .39 and below were reported as difficult. Kuder Richardson-20 and 21 (KR-20/KR- 
21) coefficients are also used in tests coded as 1-0. These coefficients are used to determine the 
reliability of the test (Baykul, 2000). The fact that the value obtained in this method is close to 
1 indicates that the reliability of the test is also high. A test analysis was conducted in terms of 
these values suggested in the literature. The data collected from the students were converted 
into .txt format and analysed with the item analysis program Test Analysis Program (TAP.exe). 


Table 2 
Evaluation Chart for pjx, rjx, and KR-20 


: AOS rjx Very good 

» 30 $ rx 39 Quite good 
.20 S$ rjxs .29 Edited and developed 
19 = rx Very weak item should be excluded from 

testing 
: 61 < pjx Easy 

PK 60<pjx< 40 Medium 
139 2 pjx Difficult 
.00 < KR-20 < .40 Unreliable 

KR-20 
40 < KR-20 < .60 Low test scores 
.60 < KR-20 < .90 Quite reliable 
.90 < KR-20 < 1.00 Highly reliable 


Research Results 


The results obtained as a result of the item analysis are presented (Table 3). Accordingly, 
the minimum score taken from the adapted test is 1 and the maximum score is 15. The mean 
score of the test is 1 and the skewness and kurtosis values show that the data have a normal 
distribution (Hair et al., 1998). The score of the discrimination levels of the items in the test 
shows that the items in the test have a very good item structure (rjx = .458). In addition, it was 
observed that the items forming the test were at medium difficulty level (pjx = .674) and the test 
was quite reliable (KR-20 = .722). In this context, split-half reliability (.613) and Spearman— 
Brown coefficients (.760) were found to have very good values. The minimum score obtained 
by the students (7 = 127) who responded to the test and were in the upper 27% for the test was 
12, and the maximum score obtained by the students in the lower 27% segment was 8. In this 
way, it is possible to conclude that the test has a valid and reliable structure. 


Table 3 

Test Analysis Results 
Examinee Results Item Results 
Number of Examinees 372 pix 674 
Total Possible Score 15 rx 458 
Min Score 1 KR-20 122 
Max Score 15 KR-21 671 
Median Score 10 High Group Min Score (n = 127) 12 
Mean Score 10 Low Group Max Score (n = 106) 8 
SD 2.970 Split-Half Reliability 613 
Skewness -.356 Spearman-Brown Coefficient 760 
Kurtosis -.299 
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The response status of each item forming the test by the students in the lower and 
upper groups, and the item difficulty and item discrimination scores were discussed (Table 
4). Accordingly, it is seen that the items forming the test were answered proportionally by the 
upper and lower groups, and there was no problematic item. However, it is seen that question 
number | in the test is a very easy item (pjx = .90) and can be edited and improved (rjx = .22). 


Table 4 

Item Analysis Results 
Item pened pjx rjx Correct in High Group (f / %) Correct in Low Group (f / %) 
Item 1 335 90 22 124 (.98) 80 (.75) 
Item 2 299 80 39 122 (.96) 61 (.58) 
Item 3 265 Aral 05 122 (.96) 43 (.41) 
Item 4 334 .90 31 126 (.99) 72 (.68) 
Item 5 158 A2 oT 93 (.73) 17 (.16) 
Item 6 198 209 61 109 (.86) 26 (.25) 
Item 7 310 83 38 124 (.98) 63 (.59) 
Item 8 314 84 35 123 (.97) 66 (.62) 
Item 9 282 16 A4 121 (.95) 54 (.51) 
Item 10 239 64 Al 105 (.83) 44 (.42) 
Item 11 295 19 36 122 (.96) 64 (.60) 
Item 12 183 AQ 64 108 (.85) 22 (.21) 
Item 13 191 51 04 100 (.79) 26 (.25) 
Item 14 193 52 65 104 (.82) 18 (.17) 
Item 15 164 44 43 85 (.67) 25 (.24) 


Finally, attempts were made to determine the effectiveness of the distracters of the items 
in the test (Table 5). Accordingly, the total answers given to each question, the answers given by 
the participants in the lower group, and the answers given by those in the upper group, and the 
difference between them are shown. When the questions in the test are considered one by one, it 
is seen that there is no more powerful distracter that can replace the answer key in the analysis 
made with the TAP.exe program. However, in some questions, it is seen that some options are 
marked by very few students. For example, 3 chose option A in the Ist question, 9 chose option 
B in the 3rd question, and 3 chose option C in the 7th question. This shows that these options 
are not very good distracters. However, this situation did not pose a problem in terms of the 
answers given by the students in the lower and upper groups to the questions. Considering the 
response rates given by those in the lower and upper groups to the questions in the test, it is seen 
that the correct answer option is marked more by those in the upper group. This shows that the 
questions in the test are appropriate in terms of discrimination. 
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Table 5 
Options Analysis 
— Group Option 1 (A) Option 3 (C) Option 4(D) 
(rjx) (rjx) (rjx) 
1 Total 30. T2 (.032) 335% (901) 
High 1 (.008) 0 (.000) 124 (.976) 
Low 2 (.019) 11 (.104) 80 (.755) 
Difference 1 (-.011) 11 (-.104) 44 (.222) 
2 Total 51 (.137) 299*(.804) 9 (.024) 
High 2 (.016) : 122 (.961) 2 (.016) 
Low 27 (.255) . 61 (.575) 6 (.057) 
Difference 25(-.239) -, 61 (.385) -A(-.041) 
3 Total 22 (.059) ; 76 (.204) 265*(.712) 
High 0 (.000) ; 5 (.039) 122 (.961) 
Low 13 (.123) ; 44 (.415) 43 (.406) 
Difference -13(-.123) : -39(-.376) 79 (.555) 
4 Total 14 (.038) j 334*(.898) 8 (.022) 
High 0 (.000) . 126 (.992) 0 (.000) 
Low 14 (.132) : 72 (.679) 7 (.066) 
Difference 14(-.132) : -54 (.313) 7(-.066) 
5 Total 158*(.425) : 77 (.207) 86 (.231) 
High 93 (.732) 8 (. 11 (.087) 15 (.118) 
Low 17 (.160) 16 (. 34 (.321) 39 (.368) 
Difference 76 (.572) -8(-. -23(-.234) -24(-.250) 
6 Total 70 (.188) 88 (. 16 (.043) 198*(.532) 
High 3 (.024) 15 (. 0 (.000) 109 (.858) 
Low 33 (.311) 37 (. 10 (.094) 26 (.245) 
Difference 30(-.288) -22(-. -10(-.094) 83 (.613) 
7 Total 310*(.833) 6 (. 3 (.008) 53 (.142) 
High 124 (.976) 1(. 0 (.000) 2 (.016) 
Low 63 (.594) 4(. 2 (.019) 37 (.349) 
Difference 61 (.382) -3(-. -2(-.019) -35(-.333) 
8 Total 12 (.032) i 11 (.030) 314*(.844) 
High 1 (.008) 2 (.016) 123 (.969) 
Low 7 (.066) 4 (.038) 66 (.623) 
Difference -6(-.058) -2(-.022) 57 (.346) 
9 Total 38 (.102) 18 (.048) 34 (.091) 
High 3 (.024) 1 (.008) 2 (.016) 
Low 18 (.170) 11 (.104) 23 (.217) 
Difference 15(-.146) 10(-.096) -21(-.201) 
10 Total 10 (.027) 28 (.075) 95 (.255) 
High 0 (.000) 1 (.008) 21 (.165) 
Low 6 (.057) 19 (.179) 37 (.349) 
Difference 6(-.057) 18(-.171) -16(-.184) 
1 Total 295*(.793) 26 (.070) 22 (.059) 
High 122 (.961) 2 (.016) 1 (.008) 
Low 64 (.604) 11 (.104) 16 (.151) 
Difference 58 (.357) 9(-.088) -15(-.143) 
12 Total 83 (.223) 66 (.177) 40 (.108) 
High 6 (.047) 9 (.071) 4 (.031) 
Low 35 (.330) 25 (.236) 24 (.226) 
Difference 29(-.283) 16(-.165) -20(-.195) 
13 Total 56 (.151) 191*(.513) 59 (.159) 
High 5 (.039) 100 (.787) 4 (.031) 
Low 27 (.255) 26 (.245) 31 (.292) 
Difference 22(-.215) 74 (.542) -27(-.261) 
14 Total 53 (.142) 32 (.086) 193*(.519) 
High 8 (.063) 1 (.008) 104 (.819 
Low 29 (.274) 18 (.170) 18 (.170) 
Difference 21(-.211) 17(-.162) 86 (.649) 
15 Total 15 (.040) 23 (.062) 170 (.457) 
High 1 (.008) 2 (.016) 39 (.307) 
Low 10 (.094) 13 (.123) 58 (.547) 
Difference 9(-.086) -11(-.107) -19(-.240) 


*is keyed answer 
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Discussion 


In the present research, TechCheck-2, a CT test developed by Relkin et al. (2020) for 
primary school students, was adapted. There are 15 questions in the test and these questions 
measure the dimensions of algorithmic thinking, modular structure, control structures, 
representation, software/hardware, and debugging associated with CT. As a result of the 
statistical analyses conducted, all items in the scale were suitable for inclusion in the Turkish 
version in accordance with the original scale. The high level of discrimination of the items in 
the test is an indication that they are successful in distinguishing students with high and low CT 
skills. In addition, the test contains items of medium difficulty that students of all ability levels 
can answer. An important condition for a test is that it is a valid and reliable scale in terms of its 
psychometric properties (Tekin, 1991). It has been determined that the test adapted within the 
scope of the present research is quite reliable; the KR-20 reliability coefficient was determined 
as .722. This coefficient is higher than the original scale's KR-20 value. This may be due to 
cultural conditions. An important issue for tests that measure ability is the comparison of the 
scores of the students in the lower 27% and upper 27% of the test. The reason for comparing 
the scores of the students in the 27% percentile is to increase the success differences in the 
normal distribution and provide sufficient examples for the analysis (Matlock-Hetzel, 1997). 
The comparison between the lower and upper groups for this test shows that it is reliable in 
terms of discrimination and that a sufficient sample is included for analysis. The strength of 
distracters is another important issue in multiple-choice test type scales (Tekin, 1991). In tests 
containing multiple-choice questions, students are required to choose between answer options, 
some of which may be partially correct. Partially true or false options are called “distracters” 
(Gierl et al., 2017) because they give partial information to students and distract the test 
solver because of the probability of being selected correctly (p.1084). In this test, which was 
adapted into Turkish, although some of the distracters did not work very well, they successfully 
distinguished the students who knew from those who did not. 


Conclusions and Implications 


As a result, it is possible to conclude that TechCheck-2, which has been adapted into 
Turkish, is a very valid and reliable tool to objectively measure children's CT skills. Tang et al. 
(2020) stated that most of the measurement tools for CT include items on specific topics such as 
computer science or coding; therefore, there is a need for tools measuring this skill in general. 
The adapted TechCheck-2 can be used for general evaluation as it contains items independent 
of the subject area. Measuring CT skills with valid and reliable tools suitable for primary school 
students will enable the students to reveal their potential in this field, and it can also be used as 
a pre-test/post-test measurement tool in determining how effective the research and educational 
content for primary school students is. By determining which dimensions of students’ skills 
are open to improvement within the scores to be obtained from this test, the relevant skill can 
be supported. It can also be useful for identifying students who need extra support and have 
special difficulties (Relkin et al., 2020). In addition, the relationships between students' literacy 
skills and CT skills can be revealed with this test. The present research was limited to the 
students who participated in the test application from the province of Ankara. In future, a wider 
sample can be included by collecting data from students from different provinces in Turkey. 
Thus, the CT development of Turkish students can be discussed by using it in both survey and 
experimental research. 
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